NASA Contractor Report 201628 

ICASE Report No. 96-71 ^, y 



ICASE 



RANK ORDERING AND POSITIVE BASES 
IN PATTERN SEARCH ALGORITHMS 



Robert Michael Lewis 
Virginia Torczon 



NASA Contract No. NAS1-I9480 
December 1996 

Institute for Computer Applications in Science and Engineering 
NASA Langley Research Center 
Hampton, VA 23681-0001 

Operated by Universities Space Research Association 




National Aeronautics and 
Space Administration 

Langley Research Center 

Hampton, Virginia 23681-0001 



f 




RANK ORDERING AND POSITIVE BASES IN 
PATTERN SEARCH ALGORITHMS* 



ROBERT MICHAEL LEWIS 

ICASE 

NASA Langley Research Center 
Hampton, VA 23681-0001 
buckaroo®icase . edu 

and 

VIRGINIA TORCZONt 

Department of Computer Science 
College of William and Mary 
Williamsburg, VA 23187-8795 



Abstract 

We present two new classes of pattern search algorithms for unconstrained min- 
imization: the rank ordered and the positive basis pattern search methods. These 
algorithms can nearly halve the worst case cost of an iteration compared to the classi- 
cal pattern search algorithms. The rank ordered pattern search methods are based on 
a heuristic for approximating the direction of steepest descent, while the positive basis 
pattern search methods are motivated by a generalization of the geometry characteris- 
tic of the patterns of the classical methods. We describe the new classes of algorithms 
and present the attendant global convergence analysis. 
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1. Introduction. In this paper we introduce two new classes of pattern search 
algorithms: the rank ordered and positive basis pattern search methods for the uncon- 
strained minimization problem 

minimize /(x). 

The rank ordered and positive basis pattern search methods extend the analysis devel- 
oped in [12]. These new classes of algorithms can almost halve the worst case cost of an 
iteration when compared with the classical pattern search methods considered in [12]. 
Moreover, the simple heuristics that motivate rank ordered and positive basis pattern 
search methods are intuitively appealing and make the methods straightforward to 
describe. 

Pattern search methods form a class of "steep descent" procedures (a term we will 
shortly explain) for nonlinear minimization. Examples include [1], [6], and [11]. While 
these algorithms have no explicit recourse to a Taylor series model of the objective, or 
any information about directional derivatives, one can develop a global convergence 
analysis for pattern search methods with results similar to those for quasi-Newton 
methods. 

This is possible because pattern search methods are gradient-related, so that when 
the steps become small enough, they are are guaranteed to capture a portion of the 

improvement promised by the steepest descent direction. This intuition seems to 
have been in the minds of the eariy developers of the broader class of direct search 
methods; for instance, Spendley, Hext, and Himsworth [9] refer to their algorithm as a 
method of "steep ascent" (they were considering maximization) to indicate its kinship 
to the method of steepest ascent. The convergence analysis for pattern search methods 
confirms that this is particularly apt and we therefore propose the term "methods of 
steep descent" to describe pattern search methods.- 

A characterization and analysis of a class of pattern search methods is presented 
in [12]. Key to the global convergence analysis is the fact that these methods produce 
iterates that lie on a rational lattice. To ensure global convergence, the pattern search 
methods discussed in [12] must satisfy a condition that prevents the mesh size from 
being reduced if one of the steps in a set of 2n core steps produces decrease in the 
objective value. This condition allows the acceptance of any step that lies in a certain 
finite subset of the lattice as long as the step produces simple decrease on the objective 
value at the current iterate, and plays the role in the global convergence analysis of 
pattern search methods that is played by the fraction of Cauchy decrease condition 
for trust-regions and the Armijo-Goldstein- Wolfe conditions for line- searches. 

As a consequence, one can construct pattern search algorithms that require, in the 
best case, only one new objective value per iteration. This feature is used to advantage 
in a proposed strategy for scientific and engineering optimization [4] for problems in 
which the computational cost of a single objective evaluation is sulficiently great as 
to merit particular care in choosing steps at which to evaluate the objective. On the 
other hand, the worst case occurs when it is necessary to reduce the mesh size to make 
further progress, for instance, because the search has reached a neighborhood either 
that contains a local minimizer or where the objective is highly nonlinear. For the 
classical pattern search algorithms studied in [12], as many as 2n objective evaluations 
may be necessary in this case. 

The rank ordered pattern search methods and positive basis pattern search meth- 
ods that we introduce here improve this worst case bound to n + 1 objective values. 
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They do this by reducing to a minimal size the set of directions that we must consider 
to be assured of having a sufficiently rich set of search directions to capture a suitably 
large component of the direction of steepest descent. 

The rank ordered pattern search methods are motivated by the following heuristic. 
Suppose that in addition to the objective value at the current iterate Xfc, we know the 
objective value at n other suitably independent points. Then the direction from the 
point with the highest objective value to the point with the lowest objective should 
be, when this pattern of points is sufficiently small, a crude estimate of the direction of 
steepest descent. Unlike a finite-diiference approximation to the direction of steepest 
descent, this approximation ignores the distances between the points (and thus the 
relative rate of change). On the other hand, this estimate only requires that we identify 
the best and worst objective values among a set of n + 1 values. This is in keeping 
with a distinctive feature of pattern search methods: namely, that they only require 
information about the relative rank of objective values, and can actually be used in 
the absence of any numerical objective value, as might be the case when the algorithm 
is driven by a subjective preference on the part of the user. 

The analysis in [12] does not make use of rank-order information, though it is 
implicit in the multidirectional search algorithm considered there. As it happens, the 
simple heuristic of using rank ordering and the notion of "steep descent", described 
above, to suggest search directions suffices to prove global convergence. We note that 
this heuristic also motivates the direct search method of Nelder and Mead, which is 
not a pattern search method and is known not to be robust [8], and the aforemen- 
tioned direct search method of Spendley, Hext, and Himsworth [9], the very interesting 
analysis of which we will discuss elsewhere. 

The heuristic of using the best and worst objective values to suggest a direction 
of steep descent reduces the cardinality of the core set of steps from 2n to n + 1, 
but may introduce an inherently sequential component into the algorithm. It may 
be the case that we cannot specify the final search direction until after the first n 
points have been evaluated and ranked. The positive basis pattern search methods 
have the attractive feature that they avoid this sequentiality and thus are well-suited 
for parallel implementation. 

It was in the course of re-examining the work on direct search methods by Yu 
Wen-ci [13, 14] that we realized the utility of the theory of positive Unear dependence 
developed by C. Davis in [3] for generalizing pattern search methods in a useful way. 
One can view the positive basis pattern search methods as the natural generalization 
of the algorithms considered in [12]. 

We say "natural" for the following reason. In the class of algorithms studied in 
[12], the 2n core directions played the technical role of ensuring that we would search 
in a direction that made a positive inner product with the direction of steepest descent. 
The notion of a positive basis [3] is the correct way to generalize this latter property, 
and allows us to reduce, a priori, the cardinality of the core set of steps from 2n to as 
few as ri + 1. Because this core set can be specified in advance, we avoid the sequential 
element of the rank ordered pattern search methods. 

Notation. We denote by R, Q, Z, and N the sets of real, rational, integer, and 
natural numbers, respectively. 

All norms are Euclidean vector norms or the associated operator norm. We define 
L{x) = {y : f{y) < f{x)}. We will denote the gradient V/(x) by g{x) and the gradient 
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at iteration k g{xk) by Qk* 

By abuse of notation, if ^ is a matrix, y^A means that the vector y is a column 
of A. We will also use c and C to denote divers constants whose identity will vary 
from place to place but whose nature will never depend on the iteration k. 

2. Pattern Search Methods. Pattern search methods, including the two new 
classes of algorithms under discussion, are characterized by the nature of the generating 
matrices and the exploratory moves algorithms, these features are discussed more 
fuUy in [12]. 

To define a pattern we need two components, a basis matrix and a generating 
matrix. We will expand the class of pattern search algorithms by expanding the class 
of admissible generating matrices. 

The basis matrix can be any nonsingular matrix B € R''^''. The generating 
matrix is a matrix Ck € Z^^^* , where p^fc > n + 1. We partition the generating matrix 
into components 

(1) Cjk - [ Tit Xfc 0 ]. 

We require that Tk € M, where M is a finite set of integral matrices with full row 
rank. In §2.1 and §2.2 we will discuss further requirements on the members of M; we 
will see that Ik must have at least n + 1 columns. The 0 in the last column of Ck is 
a single column of zeros. 

A pattern Pk is then defined by the columns of the matrix Pk = BCk- For 
convenience, we use the partition of the generating matrix Ck given in (1) to partition 
Pk as follows: 

Pk = BCk = [ BTk BLk 0 ]. 

Given A/, G R, Ajt > 0, we define a trial step s% to be any vector of the form 
s{ = AkBcl, where c\, is a column of Ck^ Note that Bc^ determines the direction of 
the step, while serves as a step length parameter. 

At iteration fc, we define a trial point as any point of the form xl = Xk + s^, where 
a;^ is the current iterate. 

Algorithm 1 states the pattern search method for unconstrained minimization. 
To define a particular pattern search method, we must specify a basis matrix B, the 
generating matrices Ck, the exploratory moves to be used to produce a step Sk, and 
the algorithms for updating Ck and Aa^. 

Algorithm 1. The pattern search method for unconstrained minimization. 
Let 0^0 e R"" and Aq > 0 be given. 
Forfc = 0,l,---, 

a) Compute f{xk)^ 

b) Determine a step Sk using an unconstrained exploratory moves algorithm. 

c) If f{xk + Sk) < f{xk), then Xk-^i = Xk + Sk. Otherwise Xk+i = Xk. 

d) Update Ck and Aj^. 

If f^xk + Sk) < fi^k) we call the iteration successful; otherwise, we caU the 
iteration unsuccessful. 

We have the following Hypotheses on Unconstrained Exploratory Moves. This is 
the same as in [12]. The specification of the matrix in (1) is different from that in 
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[12], however. There, Tk was required to be a tz x 2n matrix with full row rank. The 
Hypotheses on Unconstrained Exploratory Moves require that before an iteration is 
declared unsuccessful, we must examine all the steps in the core pattern determined 
by AkBTk for a lower objective value. Thus, for the algorithms considered in [12], in 
the worst case one would need to compute a minimum of 2n objective values in an 
iteration. In the rank ordered and positive basis pattern search methods, however, the 
size of Tk can be as small as n x (n + 1), which has the effect of reducing the worst 
case cost of an iteration to n + 1 objective evaluations. 

Hypotheses on Unconstrained Exploratory Moves. 

1. Sk e AkPk = AkBCk = Ak [BTk BLk 0]. 

2. If min { f{xk + y) \ y ^ AkBTk } < f{xk), then f{xk + Sk) < f{xk). 

Algorithm 2 specifies the rule for updating A^. The conditions on 0 and A ensure 
that 0 < ^ < 1 and > 1 for all Xi G A. Thus, if an iteration is successful it may be 
possible to increase the step length parameter A^^, but Ak is not allowed to decrease. 

Algorithm 2, Updating Aj^. 

Let T e Q, T > 1, and {wo.wi,- - ,wl} C Z, wq < 0, and ti;^ > 0, i = 1, • • Let 
^ = r^o , and e A = {r^^ , • • • , 

a) If f{xk + Sk) > f{xk) then A^+i = ^A^. 

b) If f{xk + Sk) < f{xk) then Afc+i = AfcA^. 

2.1- Rank Ordered Pattern Search Methods. As discussed in the Introduc- 
tion, one can use the best and worst objective values in a portion of the pattern to 
suggest an intuitive direction of steepest descent for exploration. This heuristic, which 
we will develop formally here, ensures that when Ak is sufficiently small, one has a 
trial step that is a suitably good direction of descent. 

For the rank ordered pattern search methods, the generating matrix is an n x pk 
matrix, pk> n + 1, which we partition as: 



We require Sk to be a nonsingular element of Z""^"". The notation Sk is chosen to 
suggest simplex, and Rk is meant to suggest reflection, as in the multidirectional 
search algorithm [11]. 

Let Sk = [d\dk-" dk] consider the simplex with vertices 



(2) {xk, xk + AkBdl Xk + AkBdl ''^,(tk + AkBd^} = {vl vl t;^, • • • , vjf }, 



Note that we only need to identify the vertices with the best and the worst objective 
values; we are not required to give a relative ranking of the remaining n - 1 vertices. 

Given the best and worst objective values among the vertices of the simplex (2), 
we can then say how Rk is chosen. We require Rk to contain a column of the form 



Ck = [ Sk Rk Lk 0 ] = [ Tk Lk 0 ]. 





Pi >Q.Pk> 0. 
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The presence of a column of this form in Rk means that the core pattern AkBTk will 
contain a trial step of the form 

n 

(5) ^A = E/^U^2-4)- 

i=l 

Since the column defined by (4) is contained in Tk = [Sk Rk], there are, implicitly, 
further restrictions on the choice of pi, i = l,-*.^? because Tk € M, and M is a 
finite set of integral matrices with full row rank. One choice would be to set Pk 0 for 
z = 1, ' • • , n - 1 and = 1. An example can be seen for in Fig. 4. Another choice 
would be to set />'^ = 1/n for i = 1, • • - ,71, provided that the Rk that results is integral. 
This can be arranged if one is willing to scale B appropriately. The latter choice gives 
equal weight to all the n potential descent directions {vl - ?;^) for z = 1, • • • , n. 

The requirement that > 0 ensures that at least one of the search directions 
defined hy BRk is biased towards the direction from the worst vertex to the best. 
This distinguished direction will, when A^; is small, capture enough of the direction 
of steepest descent to allow us to prove global convergence. 

Here we see how using the rank-order information and the steepest edge heuristic 
can introduce a sequential element into the definition of Ca?. For instance, the choice 
p[ = 0 for z = 1, ■ • • , n - 1 and = 1 means that we cannot define Rk until we have 
identified and v]^ satisfying (3). 

2,1.1. An example of a rank ordered pattern search method. In Fig. 1 
we give an example of a rank ordered pattern search method. This algorithm is 
a sequential variant of the parallel multidirectional search algorithm of Dennis and 

Torczon [5]. 

One of three possible steps are accepted at the conclusion of each iteration: a 
reflection step, an expansion step, or a shrink step. Examples for each of these three 
steps in can be seen in Figs. 2, 3, and 4, respectively. Note that since we know the 
values of f{v^)y • -,f{'^k) ^P^^ ^^^^^ ^^^^ iteration k, this fairly conservative variant 
of a rank ordered pattern search method computes exactly n + 1 objective values at 
each iteration, regardless of the step that is finally accepted. More elaborate variants, 
which attempt to compute a single objective value per iteration, are possible. The 
variant we present here has the advantage of being both simple and robust. 

2.2^ Positive Basis Pattern Search Methods. The positive basis pattern 
search methods will be described in terms of the notion of positive linear dependence 
developed in [3], Positive linear dependence captures the essential technical role played 
by the core pattern in [12]. 

The positive basis pattern search methods are also motivated by the requirements 
of parallel computing. As we have seen, the rank ordered pattern search methods may 
introduce a sequential element into the computation since we must satisfy (4). As we 
have seen, this may require the identification of the worst vertex before we can append 
the remaining search direction in the pattern. As a consequence, Amdahl's Law says 
that we can at most halve the execution time of a single iteration via computational 
parallelism, regardless of the number of processing units available. The positive basis 
pattern search methods avoid this sequentiality by imposing a geometric condition 
on Tk' Thus the positive basis pattern search methods are ideally suited for parallel 
implementation since the algebraic conditions on Tk can be satisfied a priori and 
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Start with an initial simplex with vertices v^.'-.v^} 
and evaluate f{vQ),j = 0,"', n, 
for = 0,1,--- 

Reorder the vertices of the simplex so that 

fK) < f{vl), 
Check the stopping criteria, 



2vt 



evaluate f{rk) 

if (fivk) < M)) then 

evaluate /(e^) 

if if (ek) < f{rk)) then 



for j = l,--,n- 1 



2vi 



evaluate /(^^^^i) 
end for 



else 



Tk 



for j = l,--^^- 1 

evaluate /('y^+i) 
end for 
end if 



else 



J , I ^0 



end 
end if 



fc+i 

evaluate f{vlj^-^) 



end 



/* 



r 
I* 



reflection step */ 
expansion step */ 



accept expansion */ 
expand simplex */ 



/* 



accept reflection */ 
reflect simplex */ 



/* shrink simplex */ 



Fig. 1. A straightforward rank ordered pattern search method. 



allow all the necessary objective evaluations for a single iteration to be independently 
computed in parallel. 

2.2. !• Positive linear dependence. We present here the ideas we will need 
from the theory of positive linear dependence [3]. The positive span of a set of vectors 
{ai, • • • , ar} is the cone 

{ a € R"" I a = citti + • ■ • + Crar, Ci > 0 for all i } . 

The set {ai, • • • , a^} is called positively dependent if one of the a^'s is a nonnegative 
combination of the others; otherwise the set is positively independent A positive basis 
is a positively independent set whose positive span is R'^. The following theorem from 
[3] indicates that a positive spanning set contains at least n+l vectors. 

Theorem 2.1. Suppose {cti, • • -.ar} positively spans W^. Then {a2, • • -^^r} 
early spans R". 
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Cfc 









Fig. 2. A reflection step of the straightforward rank ordered pattern search method, given the simplex 
with vertices 

The following characterizations of positive spanning sets can be found in [3] as well. 

Theorem 2.2. Suppose {ai,---,a^}, ai 0, linearly spans W , Then the follow- 
ing are equivalent: 

L {ai , • • - , ttr } positively spans . 

2. For every 6^0, there exists an i for which h^ai > 0. 

3. For every i = l,---,r, -ai is in the convex cone positively spanned by the 
remaining Ui, 

A positive basis that contains n + 1 elements is called minimal One can also show 

[3] that a positive basis can have no more than 2n elements; such a basis is called 
maximal. A maximal positive basis has a very special structure: it must consist of a 
Unear basis for R"" and the negatives of those basis vectors. On the other hand, it 
is easy to see that such a collection of 2n vectors is a positive basis, as the next two 
propositions show. 

Proposition 2.3. Suppose {ai, • • •,a^} is a positive basis for and B is a 
nonsingular nxn matrix. Then the set {Bai, • • • , Bar} is also a positive basis for R^. 

Proof Because B is nonsingular, the set {Bai,- • - ,50^} linearly spans R^. More- 
over, because {ai,--,ar} is a positive basis, by Theorem 2.2, part (2), given any 
6 G R'^, 6 7^ 0, there exists an i for which 

{B^b)'^ai = b'^Bai > 0. 

But this means that {Bai, • • • , Bar} is a positive basis for R"". □ 

Proposition 2.4. // B is a nonsingular nxn matrix^ then the columns of the 

matrix [B — B] form a positive basis. 

Proof By part (2) of Theorem 2.2, we know that the matrix [/ - /] is a positive 

basis. The result then follows from Proposition 2.3. □ 
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Fig. 3. An expansion step of the straightforward rank ordered pattern search method, given the simplex 
with vertices {vljvl^vl}. 

2.2.2, Description of the positive basis pattern search algorithm. In a 

positive basis pattern search method, we require the generating matrix Ck to be any 
nxpk matrix, pk > n + 1, of the form found in (1), but now we require Tk to be a 
positive basis for R". Proposition 2.3 then says that at every iteration k, AkBTk wiU 
also be a positive basis for R"^. 

2.2.3. Examples of positive basis pattern search methods. The classical 
pattern search methods considered in [12]— coordinate search with fixed step length, 
Hooke and Jeeves, and Evolutionary Operation using factorial designs— are all positive 
basis pattern search methods as we have defined them. The generating matrix in [12] 
has the form 

Ck = [ Mk -Mk Lk 0 ] 

for some n x n nonsingular matrix Mk^ In light of the discussion in §2.2, the nature 
of Tk — [Mk - Mk] as a maximal positive basis is now revealed. 

We are not aware of any classical pattern search method that uses a positive basis 
that is not maximal. Coordinate search with fixed step length, Hooke and Jeeves, and 
Evolutionary Operation using factorial designs all use positive bases with 2n elements. 

But it is not difficult to use the general notion of a positive basis to invent new 
pattern search methods that are proper extensions of the class of algorithms studied 
in [12]. Such methods have a practical appeal because we can reduce the number of 
objective evaluations in the worst case from 2n to as few as n + 1. 

Here are two possible choices for Tk that lead to minimal positive basis pattern 
search methods. Let e = (1, 1, • • • , 1)^, and consider 

(6) Tfc = [nl - e] ' 
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Fig. 4. A shrink step of the straightforward rank ordered pattern search method, given the simplex 
with vertices {t^fe, ^'fe}- 

and 

(7) rk = [I -e]. ' 

It is not difficult to see from Theorem 2.2, part (2), that the columns of each matrix 
form a positive spanning set, and since there are n + 1 columns in each case these 
positive spanning sets are minimal positive bases. 

Consider first the choice (6). If the basis matrix is given by 5 = ^[6i • • -bn], then 

1 

BTk = [6i • • - tn where 6n+i = -~ S^^i- 

Thus, the trial steps in this pattern are the vectors {h.-'.bn] together with the 
negative of their average. In Fig. 5 we illustrate an example of such a minimal positive 
basis for R^. 




Fig. 5. A minimal positive basis for with basis vectors bi = (4 - Xk) /or i = 1, • • • , 3. 
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For the choice (7), suppose J5 = [61 • • -ftn]* Then 



^r^: = [61 • • • 6n K+i] where 6^+1 = - 6,-. 



This pattern has an interpretation in terms of a simplex. We claim that these vectors 
are the vectors from the centroid of a simplex to the vertices of the simplex. If 
171 , — • , Vn^i are the vertices of a simplex and 



1 " • 



is the centroid, we wish to know whether we can find Vi for which bi ^ Vi — v. This is 
the same as seeking a solution of the system of equations 



where 



Let 



V - [Vi---Vn+l] 



1 T 



then we are asking whether we can solve VA 5. We have the following alternativej 
either VA = B has a solution, or there exists y such that Ay = O^By ^ 0. However, A 
and B have the same nullspace: namely, that spanned by the vector e. Thus, it must 
be the case that we can solve VA = 5, which means we can interpret the pattern as 
the vectors from the centroid of a simplex to the vertices of the simplex. 

Having noted two possible positive bases that satisfy our requirements for a pos- 
itive basis pattern search method, in Fig. 6 we give an example of a positive basis 
pattern search method. Note that this uses an exploratory moves strategy that is 
identical to that used for the Evolutionary Operation algorithm (see either [2], [10], 
or [12]). but instead of using a two-level factorial design, which requires 2"^ objective 
values per iteration, the positive basis allows us to implement a design with as few as 
n + 1 objective values per iteration. 

The relationship between positive basis and rank ordered pat- 
tern search methods* There is an overlap between the class of positive basis pattern 
search methods and rank ordered pattern search methods, as we shall soon demon- 
strate. However, it is instructive to see that the two classes are not equivalent. 

To see that we cannot necessarily cast a rank ordered pattern search method as 
a positive basis pattern search method, we return to the example given in §2.1.1, 
where {rj^ — v^) = {v^ — '^l)- Note that is a legitimate reflection step for a rank 
ordered pattern search method since it satisfies (4) and (5), Fig. 7 shows that the 

{(^fc ~ ~ ~ ^k)} ^^^^ ^ positive basis for because we 

cannot find a member of the set for which the inner product with the vector (6 - v^) 
is strictly positive. Thus we violate part (2) of Theorem 2.2. 
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Start with an initial point xq, a positive basis BT = [J57^ • "Bqf^], and Aq > 0. 
Evaluate f{xo)> 
for = 0,1,--- 

Check the stopping criteria, 

for i = 1, • • /* this loop can be parallelized */ 

4 = AkBY 

evaluate f{xk + ^l) 
end for 

Sk = argmin{/(a:A: + si)} /* the sole synchronization point */ 

if fixk + Sk) < f{xk) then 

a^fc+i =3Ck + ^k 

Afc+i = Ak 
else 

Ak+a = \^k 

end if 
end 

Fig. 6. A straightforward positive basis pattern search method. 



To see that it is not always possible to cast a positive basis pattern search method 
as a rank ordered pattern search method consider the example shown in Fig. 8. Here 
the set of vectors {{vk - v^), {vl - v^), {vl - v^)} does form a positive basis for R^ but 
the vector {vk-v^) does not constitute an acceptable reflection step for a rank ordered 
pattern search method. The difficulty in this instance is that {rk - v^) is orthogonal 
to (vl - vl), thus violating conditions (4) and (5) which our analysis imposes upon 
rank ordered pattern search methods. 

As can be seen in Fig. 9, the multidirectional search (MDS) algorithm lies in the 
intersection of the two approaches. The set of vectors {{vl - v^), • • • , (t;^ - v^), {v^ - 
vl),-'Avt'V-,)}^{{vl^vl)r-^{v]:-vl),{rl^vt),^^ 

positive basis for R"". We can implement MDS as a parallel algorithm by simultane- 
ously computing the objective values at all 2n points defined by the simplex and its 
reflection. In doing so, a priori we treat all the vertices of the original simplex, ex- 
cept the best vertex as possibly being the worst and thus remove the sequential 
bottleneck or any need to coordinate such information across processes. This is equiv- 
alent to implementing a positive basis pattern search method with a maximal positive 
basis. However, we can certainly also implement MDS as a sequential algorithm, as 
demonstrated in §2.1.1, and take full advantage of rank order information in an effort 
to reduce the number of function evaluations per iteration. We do so, however, at the 
expense of a sequential element that limits the effective use of parallel computing. 

We close by noting that the convergence analysis for pattern search methods is 
flexible enough to accommodate a myriad of other combinations that may perhaps be 
more appropriate for a given application. We have illustrated only a few of the many 
options. 

3. Convergence analysis. The convergence results for positive basis and rank 
ordered pattern search methods are like those in [12]: 
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Fig. 7. A demonstration that a rank ordered pattern search method may not necessarily be a positive 
basis pattern search method 

Theorem 3.1. Suppose L{xo) is compact and suppose f is continuously differ- 

entiable on an open neighborhood of L{xo)- Let {xk} be the sequence of iterates 
produced by either a rank ordered or a positive basis pattern search method for uncon- 
strained minimization (Algorithm 1). 
Then 

]imiiif||sf(x,)|| = 0. 

We will henceforth assume that / is continuously difFerentiable on an open set 
ft containing the compact set i(xo). However, for Theorem 3.1 (and Theorem 3.2 
to follow) we really only need / to be continuously difFerentiable on L{xq), Under 
that assumption, one would need first to derive estimates that show that if is not a 
stationary point then a pattern search method eventually wiU move into the interior of 
L{xo). One would then derive a similar set of estimates and show that these estimates 
are uniform on L{xi), in the interior of X(xo). The assumption that / is continuously 
differentiable on a set slightly larger than L{xo) makes the proof shorter and clearer 
at little expense of generality. 

As in [12], we can also obtain a stronger result. To do so, we must ultimately 
stop allowing Ajfe to become larger, and we must require that the algorithm examine 
all the points of the core pattern, according to the Strong Hypotheses on Exploratory 
Moves. 

With these restrictions, we obtain the following result: 

Theorem 3.2. Suppose L{xo) is compact and suppose f is continuously differ- 
entiable on an open neighborhood Q, of L{xo). In addition, assume that the columns 
of the generating matrices are uniformly bounded in norm, that in the update of Akj 
we have Xk = I for all k after some iteration, and that the pattern search method 
for unconstrained minimization (Algorithm 1) enforces the Strong Hypotheses on Ex- 
ploratory Moves. Finally, let {xk} be the sequence of iterates produced by either a rank 
ordered or a positive basis pattern search method for unconstrained minimization. 
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vl = (106, 56f 




t;j^= (34,-43)' 



Fig. 8. A demonstration that a positive basis pattern search meUtod may not necessarily be a rank 
ordered pattern search method 





Avl = (91,101)^ 




/ A^fc = (103, 4lf 


rl = {7,17 f 


^-^'^;^(55,29f 




V| = (19,-43)^ 



Fig. 9. A reflection step for a parallel implementation of the multidirectional search algorithm. 



Then 

Urn \\9{xk)\\ = 0. 

By restricting the update of to allow only Ajfe = 1 for all fc after some iteration, 

we are assured that limA;^+co = 0. This is a corollary of Theorem 3.18. 

The outline of the proof for Theorem 3.1 follows that of [12] as developed in [7]: 

1. First we show that given t? > 0, there exists ^ > 0 (independent of k) such 
that if II g{xk) || > V and Aa: < 6, then a pattern search algorithm wiU find 
an acceptable step without further decrease of A^. 

2. We then show that if liminf fc-^+oo ii g{^k) \\ 7^ 0, then there exists a nonzero 
lower bound on A/t- 

3. Using purely algebraic properties of iterates produced by pattern search meth- 
ods, we show that we must necessarily have liminf jt-^+oo ^k = 0. 
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strong Hypotheses on Exploratory Moves, 

1. sk e AkPk = ^kBCk = [BTk BLk 0]. 

2. If mm{f{xk + y), ye AkBTk} < f{xk), then 
f{xk 4- Sk) < min{/(a:A: + y), ye AkBTk}- 

4. We then conclude that we must have lim inf jt^+cx) || di^k) 11 = 0. 
The analysis of positive basis pattern search methods and that of rank ordered pattern 
search methods differ only in the first step. After that the two lines of analysis converge 
and are more or less identical to similar steps in [12]. 

The sections that follow correspond to the outline given above. The new results 
can be found in §3.1, where we develop the case first for the positive basis pattern 
search methods and then for the rank ordered pattern search methods. The remaining 
sections complete the analysis, relying largely on results developed in [7] and [12]. 

3.1. Existence of a direction of descent. The following proposition is the 
justification for calling pattern search algorithms "methods of steep descent." 

Proposition 3.3. There exists c > 0 such that given any t) > 0, we can find 
i/>0 such that if \\ g{xk) \\ > V (^^d Ak < there is a step 4 € AkBVk for which 



-9k 4 ><^\\9k 



4 



The proof of this result differs for positive basis pattern search methods and rank 
ordered pattern search methods and thus will be developed independently for each. 

In both cases recourse will be had to the following result, the proof of which can 
be extracted from §6.2 of [12]. 

Proposition 3.4. Suppose that {ai,---,a„} linearly spans Then given any 
a: € R^, we can find an ai for which 

where k{A) is the condition number of the matrix A = [ai • • - 0^]. 

We wiU also need the following proposition, which says that we can uiuformly 
bound the first-order Taylor series remainder. 

Proposition 3.5. Given e > Oj we can find 6 > 0 such, that if x e L{xo) and 
\\y - X \\ < 6, then 

\ f{y) - f{x) - g{x/{y - x)\< e\\y - x \\, 

Proof Let = |min(l,dist(ai(a:o),^ft)). If a: G L{xo) then the baU 

B(x,6^) = {y I l|y-a:||<^i} 

is contained in ft. Let A' be the closure of U^eL(xo)^(^? ^1); ^^^^ ^ compact by 
construction. 

Now, if X € L{xq) and y € B{x,6i) we may apply the mean- value theorem to 
obtain 
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for some z on the line segment connecting x and y. Then 

< \\9{z)-^g{x)\\\\y^x\\ 

The uniform continuity of g on K allows us to find ^2 > 0 such that if a;, y G if and 
II y - II < ^2 then 

\9{z)-g{x)\\<e. 

Thus, if we choose b = min((5i, ^2), then if a: G L{xo) and || y - x || < ^, we have 



3.1.1. The case of positive basis pattern search methods. For positive 

basis pattern search methods, Proposition 3.3 follows from purely geometric properties 
of the pattern without reference to the objective and the requirement that be 
sufficiently small. 

We begin by generalizing Proposition 3.4 to positive spanning sets. 

Proposition 3.6. Given any set {ai, • • • , a^} that positively spans ai 7^ 0 for 
i = 1, • - , r, there exists c> 0 such that for all x eBJ'f we can find an ai for which 



X ai > c II a: 



Proof We need consider only the case a: 7^ 0. According to Theorem 2.1, we can 
find a basis for R"" among the vectors {ai,---,ar}. Thus, possibly upon reordering, 
we may assume that the matrix 

A = [ai'-an] 

is invertible. Then by Proposition 3.4, given any x eR^ we can find a column ai in 
A for which 



(8) 



(9) 



T 

X a; 



> 



X ai 



K{A)^/n 

There are two possibilities to consider in (8). The first is that 

1 



x-^ai > 



there is nothing more to be done in this case. 
The other possibility is that 



II ^11 II «. II; 



—x^ai > 



1 



K{A)y/n 



x\\ II ail 



In this case, we first appeal to part (3) of Theorem 2.2 to choose a set of scalars 
fJ'jk > 0, depending only on {ai, • • • , a^}, such that for any j we can express ~aj as 

r 
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This representation for -a^ leads to 



It must then be the case that for some index £ i we have 

/...a^^a.>;:^;^ll*ll Haiti 



Since x ^ 0 we know that it must be the case that fiu > 0, so we arrive at 

x'^af > — 3——-^-^ II X II II ai 



If we let 



and 



we obtain 



fx* = max{ fijk I l<j,k<rj^k} 



a. = min(|f^ | l<j,k<r], 
I II (^k II J 



(10) ''"^-'^Th^^^'^^^^"^^- 

Combining (9) and (10) then yields the proposition. Q 

Proposition 3.3 in the case of positive basis pattern search methods now follows 
from the following stronger result. 

Proposition 3.7. For a positive basis pattern search method there exists c > 0 
such that for any fc, there is a step s)^ G ^kSTk for which 



-fifJ4>c|| Qk II 



4 



Proof At each iteration A;, BTk is a positive basis. Thus, by Proposition 3.6, 
there exists C{BTk) > 0 for which we can find v G BTk such that 

-glv>C{BTk)\\gk \\ II HI- 

However, Fjt is a member of the finite set of matrices M, so there are only finitely 
many possibilities for BTk- Taking 

c= min C(BTk) > 0 
and multiplying v by yields the proposition. □- 
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3.1 •2* The case of rank ordered pattern search methods. For rank ordered 
pattern search methods, we are only assured of a suitable descent direction when Afc 
is sufficiently small. When is sufficiently small, the relative ranking of the extreme 
objective values (best and worst or lowest and highest) determines a direction that is 
suitably close to the direction of steepest descent. 

Because the matrices Tk come from a finite set of matrices, the relative sizes of 
steps in the core pattern remain bounded and are related to Aj^: 

Proposition 3.8. There exist r,R> 0 such that for all k, if si € AkBTkf then 

vAk < II 4 II <R^k^ 

Consequently, given e > 0, there exists S > 0, independent of k, such that if 

Ak < S, then II 4 II < ^ /^^ 4 ^ Afc^Ffc. 

Proof The restrictions on Tk for both the rank ordered and the positive basis 
pattern search methods ensure that none of the columns of Tk can be zero. Since B is 
invertible, BTk can never have a zero column. Because € M and M is a finite set 
of matrices it follows that there exist a nonzero lower bound and a finite upper bound 
for the norm of aU the columns of elements of BM\ The proposition follows. 0 

If we let 7* = r/R and 7* = R/r, then Proposition 3.8 means that for any 
4,4^^Ar5rfc,wehave 



(11) 



< 



4 



Proposition 3.9 translates the rank ordering of the vertices into a statement about 
directions of descent. It says that when A^: is small enough, the edges from the 
vertices vl,-',v]^ to the best vertex are either descent directions or are not very 
steep ascent directions. 

Proposition 3.9. Given s > 0 and r/ > 0, there exists 1/ > 0 such that if 
II II > V < then 

9{vlf{vl -vl)<e\ 9{vl) \\vl-vi\ i = 1, • • • , n. 

Proof We have f{vl) - < 0 for all i = 1, • • - ,71. Applying Propositions 3.5 
and 3.8 we can find 1/ > 0 such that if A^ < u, then 

9{^lf{4 - 4) - er, II vl - 4 II < fi^l) - M) < 0- 

The result follows. □ 

The next step is to show that when Ak is sufficiently small, the direction from the 
worst vertex to the best vertex is a direction of steep descent from 

Proposition 3.10. There exists c > 0 such that given any rj > 0, we can find 
i/>0 such that if \\ giv^) \\ > v and Ak < v, then 

-9{vlf{vl - vl) > c I 9ivl) 



Proof Let 



K* = max KiBSk)-, 



where by Sk C Tk we refer to the partition Tk = [Sk Rk]' By Proposition 3.4, there is 
some index £,l<i<n, for which 



> 
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9{vl) 



vl-vi 



If Ak is sufficiently small, we can divine the sign of the inner product: by Proposi- 
tion 3.9, we can find ui > 0 such that if Ak < I'l, then 



giving - 4) < II 9iv',) II II v', - vi II 



^/n I 



This means that we must actually have 



(12) 



-g(v'kfivt-vi)> 



4 - vi 



This shows the existence of a good descent direction. Now we relate (12) to the 
distinguished direction vl - v'^. Because f{vl) < f{v'^) for all i, 

f{vt) - m) < M) - fU). 

Using this inequality and Propositions 3.5 and 3.8, given any £ > 0 we can find 1/2 > 0 
such that if Ak < 1^2 then 



giv',fiv'k - vl) - e 



vl - vl 



< f{vt) - fivl) 



< M) - M) < giv'kfivl -vi) + e 



Vk 



or 



(13) - givlfivt - vl) > -g{vl)\vl -vl)-e (| -vi\\ + 
Now choose 



vl - VI 



)■ 



s = T ^ ^ 7?mm(l,7»), 



and U2 accordingly, where 7* is as in (11). Let 1/ = min(i/i,z/2). Then, if < (12) 
holds. Meanwhile, 



(14) 



vl - Vl 



1 1 
< ^r, 



4/v*v/n 
1 1 



v1-vi 



f»?7* Vk - Vk 



< - 



1 1 



"fe A; 11 - 4n*^ 

Then (12), (13), and (14) yield 

-»(»2m-j)>|^||.(*|| 

Applying (11) again yields 



-g{vt)^{v',-vi)>j. 



1 1 



g{vl) 



4 K^y/n 



which is the desired estimate. □ 

We can now attend to the reflection step ak from (5), which may comprise aU the 
directions to the best vertex from the other vertices. 



18 



Proposition 3.11. There exists c> 0 such that given any r] > 0^ we can find 
u>0 such that if \\ g{vl) \\ > rj and Ak < i^, then 



Proof Recall that the reflection step has the form 



2=1 



SO 



(15) = -pMf [vt - - E Pigivtf {vt - 4) • 



i=l 



Using Proposition 3.10 we can find ui > 0 such that if Ajt < i/i, then 
(16) -g{vtfU-vt)>C g(vt) 



Meanwhile, given any £ > 0, Proposition 3.9 allows us to find U2 so that if Ajt < 2^2, 
then 



9ivtV{vt-vi)<e g{vt) 



for all « = 1, • ■ • , n - 1. We will choose £ felicitously in a moment. In the meantime, 
applying (11) to the previous bound yields 



(17) 

Now let u = min(i/i,i/2) and suppose Aa, < u. Returning to (15), we can apply 
(16) and (17) to obtain the bound 



-9{vtf<rk > Cpl II 9{vl) I II vl - vl II - E />l£7* || gi^l) 



> [cpl-er'Ef^kjlgivl) 



Now we choose £ > 0. Let 



and 



k 



c I . p 

e < mm 



2 7* Er=iVr 

We are assured of a nonzero and e because Tk = [Sk Rk] ^ M, and M is a finite 
set of matrices, so the minimum in each of the two preceding relations is taken only 
over a finite set. With this choice of e and /»*, we obtain 



(18) 



-9{vlf<^k>jp4 9ivl) II Wvl-v^, 
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Finally, we must relate the norm of - t;^ to that of a^. We have 
with />|. > 0, so 

(19) \\<^k\\<f:pi\\vt-^vi\\ 

Now, the vectors v^- v^.i = l,-',n are the columns of AkBSk, possibly permuted: 

where IIa: is a permutation matrix. Consider any invertible n x n matrix A with 
columns ai,---,an. For any from ej = A~^aj and = Aci we obtain the 
inequalities 1 < || A'^ || || aj \\ and || a,- || < || A ||, whence || a,- i| < k(A) || aj ||. Thus 
for any i we have 



< KiAkBSkUk) II - = ^(55^) 



The latter equality holds because Ilfc is an orthogonal transformation. Returning to 
(19), we then have 



n-l 



II II <\^pt+E PKBSk)^ 



Since there are only finitely many choices for Sk and the /)^, we can find a constant 
K, independent of fc, such that 



(20) 



(Tk II < K 



The result then foUows from (18) and (20). □ 

A consequence of the preceding proposition is that if is not a stationary point 
and if A/^ is sufficiently small, then the reflection step will improve upon the best 
objective value /(t;°), and not just f{xk)^ However, we have chosen to pose Proposition 
3.3 in terms of Xk, which is not necessarily v^. Proposition 3.3 now follows as a corollary 
of Proposition 3.11 and the uniform continuity of g: 

Corollary 3.12. There exists c > 0 such that given any t] > 0, we can find 
1/ > 0 such that if \\ g{vl) \\ > rj and Ak < v, then 



-9{xkfak>c\\g{xk) \\ \\ au \\ 



Proof, Choose i/i > 0 so small that if Ajt < z/i, then the conclusion of Proposi- 
tion 3.11 holds. Then 

-9ixkf<Tk = -9{v°kfak + (givl) - g{xk)) (7k- 

> C II g{vl) II II ak II - II 9{vt) - gixk) \\ || ak \\ . 
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Next choose 2/2 > 0 so small that if Afc < f2, then 



and 



g{4) > J 11 9{^k) II 



Then, if Ak < min(z/i, z/2), 



-9{xkf<Tk > ^ 11 9{xk) II II ^fc II 



3.2. Finding an acceptable step. From this point on the convergence analyses 
for positive basis pattern search methods and rank ordered pattern search methods 
are identical and follow more or less directly from results developed in [7] and [12]. 

The following two results come from [12], to which we refer the reader for the 
proofs. The first result indicates one sense in which Ak regulates step length. 

Lemma 3.13 (Lemma 3.1 from [12]). There exists a constant > 0, indepen- 
dent ofk, such that for any trial step ^ 0 produced by a pattern search method for 
unconstrained minimization (Algorithm 1) we have || s\. \ > C*^k' 

We also recall 

Lemma 3.14 (Lemma 3.6 from [12]). If there exists a constant C > 0 such 
that for all k, C > |l4ll. for all i = l,'-,P, then there exists a constant ^* > 0, 
independent of k, such that for any trial step si produced by a pattern search method 
for unconstrained minimization (Algorithm 1) we have Ak > '^*||4ll- 

We are now ready to state and prove the main result for this section. 

Proposition 3.15. Suppose that L{xo) is compact and that f is continuously 
differentiable on an open neighborhood Q of L{xq). Then given any r}> 0, there exists 
6 > 0, independent of k, such that if Ak < 6 and \\ g{xk) \\ > then either the 
positive basis or rank ordered pattern search method will find an acceptable step Sk; 

i.e., f{xk + Sk) < f{xk)^ 

If in addition^ the columns of the generating matrices remain bounded in norm 
and we enforce the Strong Hypotheses on Exploratory Moves, then, given any 7} > 0, 
there exist 6 > 0 and c> 0, independent ofk, such that if Ak < S and \\ g{xk) \\ > f), 
then 

f{xk+i)<f{xk)-c\\gixk)\\ \\sk\\. 

Proof. Proposition 3.3 assures us that we can find > 0 such that if Ajt < ^i, 
then there is a step 4 ^ ^kBTk for which 



(21) 



gixkfsl < -c|i g{xk) 



4 



Meanwhile, Proposition 3.5 says that we can choose ^2 > 0 such that if < ^2, then 



(22) 



f{xk + 4) - fM < 9{xkf4 + ^ri 
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Thus, if Afc < <5 = min(^i,^2), (21) and (22) yield 



f{xk + ai)-nxk)<-^\\g(xk) II II 4 II , 

and so f{xi) = f{xk + si) < f{xk) for at least one 4 € ^kBVk- The Hypotheses on 
Exploratory Moves guarantee that if 

min { J{xk + y) I ye AkBTk } < f{xk), 

then fixk + Sk) < f(xk). This proves the first part of the Proposition. 

If, in addition, we enforce the Strong Hypotheses on Exploratory Moves, then we 
actually have 

f{xk + Sk)-f{xk)<-l\\9k\\ ||4||. 
Lemma 3.13 then ensures that 

f{xk + Sk) < f{xk) - |C*A^- 11 9{xk) II . 
Applying Lemma 3.14, we arrive at 

fixk + Sk) < f{xk) - ^aV'* 11 9{xk) II II Sk II , 

which is the desired estimate. □ 

The following corollary, which follows from Proposition 3.15 and the update for 
Afc, corresponds to Proposition 3.4 in [12]: 

Corollary 3.16. Suppose that L{xo) is compact and that f is continuously 
differentiable on an open neighborhood Q of L(xo). // hminfA:-.+oo || ^(^A:) || 0, 
then there exists a constant A* > 0 such that for all k, A^^ > A*. 
If liminfA;-.+oo || 9{xk) \\ ^ 0, then Proposition 3.15 says that there is a uniform 
bound ^ > 0 such that once Aat < the pattern search algorithm will necessarily find 
an acceptable step. Since we reduce Ajt only if we have an unsiiccessful iteration, this 
means we would at some point stop reducing Ajt- 

3.3, The algebraic nature of the iterates. The iterates produced by pattern 
search methods have a specific algebraic form. The rank ordered and positive basis 
pattern search methods inherit this basic algebraic structure. The next result, a proof 
of which can be found in [12], is key to the convergence of pattern search methods. 

Theorem 3.17 (Theorem 3.2 from [12]). Any iterate produced by a pattern 
search method for unconstrained minimization (Algorithm 1) can be expressed in the 
following form: 

XN=^xo+ {P'^^a"''^^) AoB J2 

k=0 

where 

• xq is the initial guess, 

• P/a = T, with a,/? G N and relatively prime, and r is as defined in the 
algorithm for updating Ak (Algorithm 2), 

• TLB cLnd ruB depend on N , 
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• Ao is the initial choice for the step length control parameter^ 

• B is the basis matrix^ and 

The next theorem combines the strict algebraic structure of the iterates, the simple 
decrease condition, and the algorithm for updating Ajt, to reach a conclusion about 
the limiting behavior of Ajb. 

Theorem 3.18. Suppose that L(xo) is compact. Then lim inf ib-H.+oo = 0. 
The proof is identical to that of Theorem 3.3 in [12]. Briefly, Theorem 3,17 says that 
the iterates lie on a rational lattice. If liminfA;^+oo 7^ 0, then there can be only a 
finite number of distinct points visited by the algorithm. However, Proposition 3.15 
says that unless we are at a stationary point, we will eventually choose a new iterate 
with a strictly lower objective value; this is at odds with the fact that we only visit a 
finite number of distinct points. 

3.4. The proofs of Theorem 3.1 and Theorem 3.2. The conclusion of the 
proof of Theorem 3.1 is identical to that of Theorem 3.5 in [12]. Suppose that 
liminffc-^+cx) |1 9{xk) \\ ^ 0. Then Corollary 3.16 tells us that there exists A* > 0 
such that for all k, Ak > A*. But this contradicts Theorem 3.18. 

The proof of Theorem 3.2 follows that of Theorem 3.7 in [12], to which we refer 
the reader. 

4. Concluding remarks. We believe that versions of the two classes of algo- 
rithms we have introduced in this paper can be developed for bound constrained 
minimization as well. The positive basis pattern search methods for unconstrained 
minimization require a positive basis for R""; the correct analog for bound constrained 
minimization should require a positive basis for the tangent cone of the feasible region 
at each iterate. The work in [7] uses such a basis that appears to be maximal in 
size. The theory of positive linear dependence used in the unconstrained case suggests 
a line of development to sharpen the results in [7], which may in turn reduce the 
computational cost per iteration in the bound constrained case. 

Similarly, the heuristic of approximating the direction of steepest descent using 
the best and worst of n + 1 vertices (determined by their objective values) should also 
be applicable to the bound constrained case. There we would only consider feasible 
points and develop a crude approximation for the direction considered in gradient 
projection algorithms. 
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